Support for Databricks deletion vectors by ianton-ru · Pull Request #2271 · Altinity/ClickHouse

ianton-ru · 2026-08-26T15:08:03Z

Changelog category (leave one):

New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Support for Databricks deletion vectors

Documentation entry for user-facing changes

Databricks put deletion vector into .bin file instead of puffin files.

CI/CD Options

Exclude tests:

Regression jobs to run:

Databricks writes the spec-aligned deletion-vector-v1 envelope in a Delta container, so Iceberg reads failed on Puffin magic. Decode that slice from content_offset when the object is not a Puffin file. Co-authored-by: Cursor <cursoragent@cursor.com>

Spark Iceberg DELETE writes Puffin, so convert those objects to the Databricks UniForm layout before the ClickHouse read. Co-authored-by: Cursor <cursoragent@cursor.com>

…ctors Spark Iceberg DELETE writes Puffin; this customer table stores DVs as version-1 `.bin` objects with `content_offset = 1`. Co-authored-by: Cursor <cursoragent@cursor.com>

The root gitignore drops `*.bin`, so the previous commit omitted the object the test actually reads. Co-authored-by: Cursor <cursoragent@cursor.com>

github-actions · 2026-08-26T17:38:23Z

Workflow [PR], commit [d8e94ed]

ianton-ru · 2026-08-26T21:06:13Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d8e94ed1bb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-08-26T21:24:17Z

+        if (isDeletionVectorV1Envelope(envelope, content_size_in_bytes))
+            return IcebergDeletionVectorContainer::SliceOnly;


Reject unknown wrappers before selecting the slice-only path

When a Puffin header is damaged—or a .bin object uses an unknown version—this accepts any valid envelope found at the manifest offset as SliceOnly. That skips bindDeletionVectorBlob, including the footer checks for the referenced data file, fields, and cardinality, so malformed input can silently filter rows instead of being rejected. Require either the known Delta 0x01 container layout or an exact bare-envelope layout before taking this path.

AGENTS.md reference: AGENTS.md:L153-L153

Useful? React with 👍 / 👎.

I did not find strong description what can and can't be in non-puffin DV files. So prefer to accept all, until report about specific files that must be declined.

ianton-ru · 2026-08-27T15:32:01Z

@blau-ai

blau-ai · 2026-08-27T15:39:09Z

CI triage — @ianton-ru

Verdict: 0 of the failures are caused by this PR. They break down as 3 flaky/infra stateless tests (all in non-default storage configs), 2 integration shards killed by the 3-hour xdist session-timeout (with only pre-existing flaky tests failing inside them), and 1 base-image CVE scan. The PR's own feature is healthy: all 36 test_deletion_vectors.py cases PASSED (incl. every test_deletion_vectors_delta_bin[...]), and other Iceberg-read tests (test_minmax_pruning, test_expire_snapshots, …) passed on the same build.

Head SHA analyzed: d8e94ed.

Per-failure breakdown

Check	Failing test	Cause	PR-caused?
Stateless (amd_debug, distributed plan, s3)	`03572_export_merge_tree_part_limits_and_table_functions`	Flaky — CI's own diagnosis: "All reruns passed (6/6), not reproducible". S3/hive export, not Iceberg.	❌ No
Stateless (amd_binary, cas s3)	`02265_column_ttl`	Flaky (7/10 reruns failed) — `NETWORK_ERROR` on manifest relink in the experimental cas s3 replication path. Column-TTL test, not Iceberg.	❌ No
Stateless (amd_asan_ubsan, cas s3)	`03578_merges_with_and_without_ttl`	Infra — `S3Exception … Code 111, Connection refused` writing to the `cas_s3` bucket. TTL merges, not Iceberg.	❌ No
Integration 4/8 (asan_ubsan, db disk, old analyzer)	`test_schema_inference` ×12	Shard hit `session-timeout: 10800s exceeded`. The 12 `test_schema_inference` cases failed en-masse in 40s with instant `ATTEMPT_TO_READ_AFTER_EOF` — the module's server instance came up unhealthy (log shows `Can't connect to Minio … Connection refused` during its cluster startup) on an overloaded runner. Isolated to that instance; the other worker kept passing Iceberg tests.	❌ No
Integration 5/8 (asan_ubsan, db disk, old analyzer)	`test_rabbitmq_select[0/1]`, `test_rabbitmq_csv_with_delimiter`	Same `session-timeout: 10800s exceeded`. RabbitMQ tests failed with `Failed: Time…` (known-flaky/slow RabbitMQ), no Iceberg involvement.	❌ No
GrypeScanServer (-alpine)	image CVE scan	7 high/critical CVEs in the alpine base image (apk/musl packages). The non-alpine `clickhouse-server` and `clickhouse-keeper` images scanned 0 vulns. Nothing in this C++ change touches it.	❌ No

Why the two integration shards are the systemic issue (not the PR)

Both amd_asan_ubsan, db disk, old analyzer shards run a huge per-shard test list across only 2 workers with a 3-hour cap (-n 2 --dist=loadfile --session-timeout=10800) and both blew that budget:

4/8: !!!!!! xdist.dsession.Interrupted: session-timeout: 10800.0 sec exceeded !!!!!!!
5/8: !!!!!! xdist.dsession.Interrupted: session-timeout: 10800.0 sec exceeded !!!!!!!

Shard 5/8 contains none of this PR's new tests and still timed out — so the timeout is a capacity/sharding problem, not something the added tests introduced. The PR's Iceberg tests that did run were quick (test_deletion_vectors.py finished 10:30→10:38, ~8 min) and all green.

Suggested next steps

No code change is needed for CI here — nothing points at the diff.
Re-run the failed jobs (Integration tests 4/8 & 5/8, and the 3 stateless suites). The stateless ones are flaky/infra and the integration ones just need to fit under the 3h window; a fresh run on a less-loaded runner should go green. The Grype-alpine finding is orthogonal to this PR (base-image CVEs) and shouldn't gate it.
⚠️ One coverage gap to note: because shard 4/8 timed out, your new test_deletion_vectors_databricks_uniform.py was scheduled but never actually executed (it appears in the shard's collection list but produced no result). test_deletion_vectors.py covered the .bin path via test_deletion_vectors_delta_bin[...] and passed, but the dedicated Databricks-UniForm test still needs a green run to be sure it's covered. Worth confirming it runs on the re-triggered job.

_{Automated triage from CI reports (praktika result_pr.json + shard job.logs). I can't build/run ClickHouse in this environment, so correctness is asserted from CI evidence, not a local run.}

ianton-ru and others added 4 commits August 26, 2026 15:27

Add Iceberg integration coverage for Delta .bin deletion vectors

60be0ab

Spark Iceberg DELETE writes Puffin, so convert those objects to the Databricks UniForm layout before the ClickHouse read. Co-authored-by: Cursor <cursoragent@cursor.com>

Add a Databricks UniForm Iceberg fixture for Delta .bin deletion ve…

4654495

…ctors Spark Iceberg DELETE writes Puffin; this customer table stores DVs as version-1 `.bin` objects with `content_offset = 1`. Co-authored-by: Cursor <cursoragent@cursor.com>

Track the Databricks UniForm deletion-vector .bin fixture

d8e94ed

The root gitignore drops `*.bin`, so the previous commit omitted the object the test actually reads. Co-authored-by: Cursor <cursoragent@cursor.com>

chatgpt-codex-connector Bot reviewed Aug 26, 2026

View reviewed changes

ianton-ru marked this pull request as ready for review August 27, 2026 15:44

ianton-ru added antalya antalya-26.6 labels Aug 27, 2026

mkmkme approved these changes Aug 28, 2026

View reviewed changes

Navigation Menu

Uh oh!

Support for Databricks deletion vectors - #2271

Support for Databricks deletion vectors#2271
ianton-ru wants to merge 4 commits into
antalya-26.6from
iceberg-delta-bin-deletion-vectors

ianton-ru commented Aug 26, 2026

Uh oh!

github-actions Bot commented Aug 26, 2026

Uh oh!

ianton-ru commented Aug 26, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Aug 26, 2026

Uh oh!

ianton-ru Aug 27, 2026

Uh oh!

ianton-ru commented Aug 27, 2026

Uh oh!

blau-ai commented Aug 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		if (isDeletionVectorV1Envelope(envelope, content_size_in_bytes))
		return IcebergDeletionVectorContainer::SliceOnly;

Uh oh!

Conversation

ianton-ru commented Aug 26, 2026

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

CI/CD Options

Exclude tests:

Regression jobs to run:

Uh oh!

github-actions Bot commented Aug 26, 2026

Uh oh!

ianton-ru commented Aug 26, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

ianton-ru Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

ianton-ru commented Aug 27, 2026

Uh oh!

blau-ai commented Aug 27, 2026

CI triage — @ianton-ru

Per-failure breakdown

Why the two integration shards are the systemic issue (not the PR)

Suggested next steps

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants